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In this paper we describe the design and implementation of a technology-active 
introductory first year university mathematics course. The design principles 
underpinning the course are presented. The results of the implementation show some 
areas where the technology-active approach has proven of value, as well as 
improvements that can be made for the next cycle. Some implications for the 
integration of technology in large lecture undergraduate teaching are presented. 


BACKGROUND 


This paper describes the implementation of a study comprising one of three 
components of a wider research project, led by a research team at the University of 
Auckland, entitled Capturing Learning in Undergraduate Mathematics. The Intensive 
Technology Innovation reported here investigates a digital technology initiative in an 
entry-level mathematics course where technology is employed in four major ways, as 
described in the course design principles below. 


The use of technology in undergraduate mathematics is well-established with respect 
to lecturer use of mathematical environments (Thomas & Holton, 2003; Drijvers, 
2012), and students who have used technology within components of particular 
courses, for example as laboratory assignments (Oates, 2011). However, it is less 
common for students to use technology as intensively as in this study, especially in the 
sense of having integrated, unrestricted use of mathematical environments and 
websites (such as Matlab & Wolfram Alpha) in \ectures, tutorials and assessments, and 
being able to access these ubiquitously (in all coursework except the final exam) 
through smart-phones or portable computers (Hoyles & Lagrange, 2010; Oates, 2011). 
Such technology use in tertiary education is strongly indicated (Hoyles & Lagrange, 
2010; Stewart, Thomas, & Hannah, 2005). Potential benefits include increased student 
engagement in mathematical activities and discourse (Scucuglia, 2006); improved 
inter-representational versatility (Thomas & Holton, 2003); and improved 
understanding in particular content areas (Thomas & Holton, 2003, Oates 2011). There 
are also studies that describe potential difficulties with the use of technology (e.g. see 
Drijvers, 2012), and Oates (2011) identifies a complex range of factors that should be 
considered to achieve an effective and integrated technology-active learning 
environment. For example, Gyéngy6si, Solovej and Winslow (2011) found evidence 
that weaker students commit more errors when technology is present, while Stewart, 
Thomas and Hannah, (2005) note that students need time for instrumental genesis 
(Artigue, 2002). Particular factors considered in the design of this study include 
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teacher privileging (Kendal & Stacey, 2001); access to technology and congruence 
between learning and assessment (Oates, 2011); student instrumentation (Stewart, 
Thomas, & Hannah, 2005); technology-active, -neutral and —trivial assessment (Oates, 
2011); students’ use of lecture recordings (Yoon, Oates & Sneddon, 2013); and the 
pragmatic, pedagogic and epistemic value of technology for particular topics in the 
curriculum (Artigue, 2002; Stacey, 2003). 


METHODOLOGY 


This research follows a design experiment methodology where “a primary goal for a 
design experiment is to improve the initial design by testing and revising conjectures as 
informed by ongoing analysis of both the students’ reasoning and the learning 
environment”. (Cobb, Confrey, diSessa, Lehrer, & Schauble, 2003, p. 11). The focus 
of this study is a technology intervention introduced to a university mathematics 
course during the second semester of 2013. The research questions ask if the course 
design was effective in engaging students in learning mathematics, and if so how, and 
what improvements might be made for future cycles. What we present here are some 
results from the initial cycle in this continuing experimental process. 


Course design principles 


There are at least four guiding principles employed in the initial cycle of course design 
and construction. First, technology should be integral to the assessment process. 
Hence, each student was required to register and enrol into MathXL — a web-based 
homework, tutorial and assessment system, which was used for five skills quizzes (1% 
each) and the mid semester test (10%). The MathxXL program provides instant feedback 
by marking student answers, identifies topics where the student needs to focus their 
attention and directs them to sections in an online textbook as well as creating a 
personalised Study Plan. The quiz and test questions were largely free-response, 
exercising the MathXL facility for numerical, algebraic and graphical input of 
solutions, in contrast to static multiple-choice style questions. The quizzes are a 
time-limited, non-supervised assessment where students have three attempts and their 
best score is recorded. The mid-semester test is also time-limited but held in a 
supervised computer lab with one attempt per question. Students were allowed access 
to CAS-calculators if they had them as well as online resources, although time factors 
would have made this impractical for most. There is still some debate about whether 
the test should be technology-free (skills-based) or technology-active. 


The second principle was that the lecturers should model a range of appropriate 
technology including: a web-based graphing calculator; YouTube clips; applets to 
demonstrate critical features of mathematics; and mathematical websites. In addition 
to the importance of teacher-privileging (Kendall & Stacey, 2001), another reason for 
this was to minimise any disadvantage to students who did not have access to specific 
technologies. At the end of each lecture students were directed to webpages that 
illustrated the concepts at the heart of each topic, and a video-recording of each lecture 
was available to students within 24 hours. Third, students were encouraged to use any 
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technology platform they had access to, including all calculators, mobile phones, 
computers, tablets, etc. and any e-resources they could access with these. The final 
design principle was that technology should be actively used in the one-hour weekly 
tutorials that all students were expected to attend. 


The course consists of 36 one-hour lectures and 10 one-hour tutorials, which are worth 
8% of the final grade. At the end of the course students were asked to complete three 
questionnaires: a technology questionnaire; an attitude survey; and the standard 
university student course evaluation. Three volunteers also worked on 
technology-active group tutorial tasks held in a computer lab. As they worked on these 
tasks field notes were made on an observation schedule. Each of these students had 
their own CAS-calculators and could use any technologies they thought appropriate. 
Figure 1 shows examples of the questions used in the online technology questionnaire, 
which contained a mix of 19 open and closed questions, and investigated student use of 
technology in general; mathematics-focused technology use; and the student pattern of 
technology use during the course. The open questions had unlimited response space. 


1. Which mathematical-learning technologies did you observe the lecturers or tutors using and modeling in 
their teaching of MATHS 102? Please select all that are appropriate. 


MathxL WU Graphics or CAS calculators U Autograph ( Wolfram Alpha U 
GeoGebra U Khan Academy UW Smartphone or Tablet App O 
Other Internet Use (specify) U Other Technology (specify) OI" 


2. Which mathematics learning technologies did you personally use in the course? Please indicate your 
frequency of use, and whether this was the first time you had used them. 


Often = almost daily; Sometimes = 1 or 2 times per week; Seldom = a few times in the semester 
MathxXL OftenO Sometimes O Seldom O- Never O 


7. What activities did you use technology for? Please specify which technologies you used for each of the 
following activities: [Lectures, assignments, tutorials, quizzes] 


11. Describe the kind of activities you used technology for when working on mathematics problems in the 
course. 


Figure 1: Examples of the open and closed questions from the questionnaire. 


Attitude to learning mathematics with technology Suggested goals 

I like using technology to learn maths My primary intention in using technology in maths 
is to check my work 

Using technology in maths is worth the extra effort My main purpose in using technology is to get the 
answer to the problem I’m working on 


Maths is more interesting when using technology When I use technology I aim to finish as soon as 
possible 

Using technology hinders my ability to understand My main goal in using the technology is to get a 

maths better grade in the course 

I prefer working out maths by hand rather than using I use the technology to find more than just the basic 

technology answer to the question 


Table 1: Examples of the scale items 


For the attitude survey, a Likert scale was constructed with five subscales in 29 
randomised items and a range of five possible responses (strongly agree, agree, neutral, 
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disagree, and strongly disagree). The subscales measured: attitude to maths ability; 
confidence with technology; attitude to instrumental genesis of technology (learning 
how to use it); attitude to learning mathematics with technology; and attitude to 
versatile use of technology. The versatility subscale had four questions and the others 
five. In addition, there were five questions covering possible goals in technology use, 
which was not a subscale. Table 1 gives examples of some items. 


RESULTS 


22 students (out of 131 in the course who sat the final exam) participated in the study; 
thirteen of these completed the questionnaire and nine the attitude survey. Responses 
were anonymous so it was not possible to tell how many were in the intersection of the 
two groups. Although this is a relatively small number of responses, we still believe it 
gives a reasonable indication of the student reaction to the course. In addition, 50 
students completed the online course evaluation. In the questionnaire, ten of the 13 
students (76.9%) agreed that the lecturers had made sufficient use of the technologies 
in the lectures, and recognised the use of a range of platforms. They agreed there was a 
wide use of technology during the course. All used MathXL, seven almost daily and six 
once or twice a week; 11 used Desmos, six of them daily, two once or twice a week; 
and six used Wolfram Alpha, five of them daily. Khan Academy was used daily by five 
students, Autograph by two and GeoGebra by one. In addition ten students made daily 
use of a graphic or CAS calculator. 


All the students used MathXL for the assessment quizzes, at least once or twice a week, 
with a mean of 4.72 out of five quizzes. Similarly, all used it for homework, ten at least 
once or twice a week and twelve for revision, ten at least once or twice a week. 
Furthermore, nine used it in their study plan and ten for help with solving problems, 
mostly at least once or twice a week. For the assignments, along with various internet 
sites, six students mentioned using calculators, five Desmos, five Khan Academy and 
two each Autograph and MathXxXL. Nearly all the students owned a laptop (12) and a 
smartphone (11), with ten also having a home computer and four a tablet. Nine (69.2%) 
had external access to Desmos, five (41.7%) to Autograph and two (16.7%) to 
GeoGebra. On average they found Desmos useful (3.9 out of 5), Autograph slightly 
useful (3.33) and GeoGebra not useful (2.2). In response to the summary questions, 
twelve (92.3%) said that they thought the technology use had helped their learning of 
mathematics, eleven (84.6%) liked the extensive use of the technology and twelve 
(92.3%) wanted the technology to be available in future courses. Some comments they 
made included: 


I learnt a lot from this course through the many technologies made available to me. I spent 
several hours each week practicing using various websites, apps and online tutorials, as 
well as recorded lectures. Highly recommended. 


MathXL helped me to focus on areas of maths I needed help with. 
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There was a broad use of mathematical technology throughout this course, enabling 
students to feel supported in the learning process...technology (for) visual learners like 
myself (makes) maths seems less daunting. 


Particularly in year one mathematics, the use of technology has helped me gain a quicker 
and deeper understanding as to how various equations behave and being able to quickly 
look up a mathematics problem on the internet also assisted greatly. 


[It should be used in future] Because it is really useful for understanding concepts, for 
practising them and learning them. 


25 of the 42 open responses to the course evaluation item “What was most helpful for 
your learning?” specifically cited technology, with positive references to MathXL and 
the quizzes (17), recorded lectures (7) and access to the web (8), for example Desmos, 
Khan Academy and WolframAlpha. Comments included: 


Mainly the recorded lectures — I've found them very useful for going over when I haven't 
understood something or forgotten something. 


MathXL was extremely helpful for my learning. Being able to check my answers instantly 
was a great encouragement and stimulant. 


MathxXL: The website was amazing — the instant feedback on answers and also the facilities 
to learn what I did wrong, as well as how to do it correctly were fantastic. 


Being prompted during lectures of other sources of information available such as Desmos 
and Khan Academy, to be able to be used concurrently with MathXL's resources. 


I think the quizzes online are the best method for cementing your knowledge of the math. 


Data from the MathxXL website and the lecture recording access also support a high 
level of student engagement with the technology. While we would expect a high 
proportion of students to access the quizzes and the test because they are assessed 
(average of 95 across the 5 quizzes; 124 for the test out of 130 students), a significant 
number of students still engaged with revision exercises and individual study plans 
(e.g. 93 & 84 respectively for the test revision and 41 for an exercise on 
differentiation). Similarly, the lecture recordings were well used, with an average of 
more than 100 student-accesses to each individual lecture, and a peak of more than 200 
for two lectures, one on logs and exponential functions and one on trigonometry. 
However, not all comments and experiences were positive. Two students in the 
questionnaire presented forceful reasons for a negative perspective: 


MathXL was a disastrously unfair method of assessment as it was difficult to formulate 
your thoughts when a test is in such a different format to what you have always done. I 
have personally always been rather good at maths but I have done very poorly in this 
course as I have struggled with everything being computer/technology based. 


... too reliant on technology without understanding the core foundations of mathematics. It 
is like designing a bridge without first knowing fundamental engineering principles. 


These sentiments were echoed in a few responses to the course evaluation item “What 
improvements would you like to see?’”’, where comments were mostly about syntax or 
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the use of MathxXL in assessment, for example: “I didn't like using Math XL for the 
mid-term test.” and “I also strongly disliked MathxXL, for multiple reasons, and I lost 
marks on a few questions in quizzes for incorrectly entering something rather than for 
getting the wrong answer.” The means of the attitude survey subscales, shown in Table 
2, indicate student agreement that they have a positive attitude to their maths ability, to 
learning maths with technology and to versatile use of technology. The level of 
agreement rises in terms of their confidence in technology use and their attitude to 
learning how to use the technology (instrumental genesis). 


Subscale Mean* (Low-High) Cronbach Alpha 

Attitude to maths ability 3.89 (3.33-4.56) 0.695 
Confidence with technology 4.42 (4.33-4.44) 0.910 
Attitude to instrumental genesis 4.40 (4.11-4.56) 0.820 
Attitude to learning 3.93 (3.11-4.22) 0.838 
mathematics with technology 

Attitude to versatile use of 4.11 (3.67-4.44) 0.872 
technology 


*Scores on negative items were reversed. 5 represents Strongly Agree. 
Table 2: The means of the subscale responses and the reliability measures. 


To gauge the internal consistency of the subscales the Cronbach Alpha (CA) measure 
of reliability was calculated. Four of the subscales show good or excellent reliability. 
The consistency of the subscale Attitude to maths ability is marginal (a CA of 0.7 is 
considered acceptable) but the CA would rise to 0.801 if the item ‘I can get good 
results in maths’ (mean = 3.33) were excluded. This may indicate that even those who 
see themselves as good at maths may be less confident of getting good results. The 
levels of agreement with the suggested goals for technology use were: My primary 
intention in using technology in maths is to check my work (4.00); My main purpose in 
using technology is to get the answer to the problem I’m working on (3.11); When I 
use technology I aim to finish as soon as possible (2.78); My main goal in using the 
technology is to get a better grade in the course (3.56); and I use the technology to find 
more than just the basic answer to the question (4.11). So while students are using 
technology to check their by-hand work it is often not just a basic answer. They are 
mostly neutral on whether they only use the technology for their current task or 
whether they try to finish as soon as they can, and probably do want to use the 
technology to get a better course grade. In Q16 of the questionnaire the students were 
asked ‘Describe what you see as your main goals in using technology in the course’, 
with space for up to three goals. Without any suggestions to lead them, nine students 
contributed 20 goals, most commonly: to improve learning and understanding of 
mathematics (6); to apply the mathematics, especially in the real world (3); and to 
practise mathematics (2). 


Data from the group of three students working on the specially designed 
technology-active tasks has yet to be fully analysed. While space prohibits reproducing 
the tasks here, the tasks were designed with two main purposes and goals in mind: 
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firstly, they were non-directed problems to be worked on as a group; and secondly to 
facilitate and encourage active use of technology. All three volunteers were clearly 
enthusiastic about the use of technology; all had their own CAS-calculators, and made 
frequent use of the computer while working on the tasks. They were all enthused about 
the online graphing package Desmos especially its free availability and ease of use. An 
interesting observation came in the second tutorial, where one of the students, after 
they had effectively answered the set questions, used the internet to explore the nature 
of their findings (fitting a polynomial curve through a number of points). One negative 
observation was that on several occasions one or more of the students became 
disengaged from the group to work individually on their calculators, although the 
computer acted more as a focal point for the group. 


The examination results at the end of the course showed a pass rate of 74.6% with 
23.1% A grades, which compared well with previous corresponding semesters when 
technology was not integrated, such as 2012 (76.6%, 21.0%) and 2011 (77.4%, 
26.4%). Thus the students were not disadvantaged by the course changes in terms of 
results. The student course evaluation, completed by 50 students, confirmed 
satisfaction with the course, with 77.1% satisfied overall with the course quality. 


DISCUSSION 


The evidence from this first implementation of the technology-active undergraduate 
mathematics course supports the value of this kind of intervention. In particular, most 
of the students enjoyed the experience, especially the use of MathXL for revision and 
quizzes, and were highly engaged with the mathematics through the technology. Their 
confidence in using the technology and attitudes to technology use of all kinds, and, 
importantly, to learning mathematics through the technology, were all very positive. 
The examination results confirm that the effect on assessed learning was at worst 
neutral, with clear indications that the technology had both pragmatic and epistemic 
value (Artigue, 2002) in facilitating understanding. There were two factors that appear 
to have significantly enhanced student engagement, as suggested by Scucuglia (2006). 
One was the relative ease of instrumental genesis of some of the technology, especially 
the Desmos program. The second was the crucial role of lecturer example, privileging 
the use of the technology in learning (Kendal & Stacey, 2001). This was not only noted 
and commented on by students but seems to have led to a wider and increased level of 
participation in technology use. We have learned that the attitude scale used is robust 
and reliable, with a minor adjustment needed to one subscale. Other lessons include the 
need to increase student participation (especially in the surveys and collaborative 
tutorials), providing information and requesting volunteers early in the course, and 
scheduling interviews earlier too. The positive outcomes described here, and the 
lessons learned from this implementation, will be taken forward into a second cycle of 
the course in semester one 2014. Integration of an intensive, technology-active 
intervention in a large undergraduate mathematics class is relatively rare. This research 
has demonstrated that implementing such a programme is not only feasible and can be 
done smoothly, free of problems, but also that it has considerable potential benefits. 
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